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ABSTRACT 

One of a series of eiiteen kncwledge transformation 
papers^ this paper represents a coffiprehensive study of the nature and 
types of impact evaluaticn available to vocatlcnal administrators, 
researchers, and evaluation experts. After explaining the concepts of 
the CIPP (context, input, process, product) fficdel, of summative 
evaluation and the value of using coinparison groups, of efficiency 
vs, ef fectlviness, of intrinsic vs* pay-aff evaluation, and of impact 
evaluation vs* research, the first section concludes wilOi a working 
definiticn of impact evaluation. The next section discusses the 
importance of this type of evaluaticn, especially in relation to 
federal legislation which requires data collection and reporting, 
planning, and evaluation to provide accountability. Next, current 
evaluaticn practices are reviewed to show their strengths and 
weaknesses. On-site visitation, evaluation of student competencies, 
and fcllcw-up evaluation are analyzed in depths Ihe fourth section 
focuses on current developments such as evaluation models for states, 
a vocational tducation study conducted by the National Institute of 
Educaticii, the role of the National Center for Research in Vocational 
Iducation, and various nitional surveys like the National 
Lcngitudinal Surveys* Following a prognosis for the future of impact 
evaluaticn, the paper ends with a lengthy biblicgraphy, (BLG) 
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FOREWORD 



Program improvement and accountability^ demands placed on vocational education 
programs and personnel make the topic of impact evaluation extremely timely, 
'rhis paper represents a comprehensive study of the nature and types of impact 
evaluation available to vocational administrators, researchers, and evaluation 
experts. It Includes a working definition of impact evaluation applicable to 
vocational education, a discussion of reasons for the importance of this type 
of evaluation, an analysis of existing models, a description of developmeiits 
in this area, and a prognosis, 

"Impact Evaluation in Vocational Education: The State of the Art" is one of 
16 papers produced during the first year of the National Center's knowledge 
transformation program;. The 16 papers are concentrated in the four theme 
areas emphasized under the National Center contract: special needs subpopula- 
tions, sex fairness, planning, and evaluation in vocational education. The 
review and synthesis of research in each topic area ir intended to coiTmunicate 
knowledge and suggest applications. Papers should be of interest to all 
vocational educators, including administrators, researchers, federal agency 
personnel, and the National Center staff. 

The profession is indebted to Dr, John T. Grasso for his scholarship in pre- 
paring the paper. Recognition is also due Dr. Donald L, Clark, Texas A 5 M 
University/ Dr, James WalU' Mississippi State University, and Dr. William 
Hull, the National Center for Research in Vocatidnal Education, for their 
critical review of the manuscript, Dr, CarolP, Kowle, research specialist, 
supervised the publication of the series. Ms, Jo-Ann Cherry coordinated 
editing and production, 

Robert Taylor 
Executive Director 
National Center for Research 
in Vocational Education 
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INTRODUCTION 



Professionals in vocational education need to be knowledgeable about thti 
strengths and weaknosscs of evaluation. Even if they do not participate di- 
rectly in evaluationss they can offer support to those engaged in such work in 
their local areas or elsewhere, and recognize opportunities for using relevant 
findings in their programs. Findings can be used for effective decision makings 
and for informing students, parents, taxpayers, and legislative representatives 
about the effect of vocational education. 

Impact evaluation is an emerging type of evaluation-research that can be used 
in vocational education for achieving the highest quality programs. This type 
of evaluation can be used to document the effectiveness of existing programs, 
to suggest improvements, and to satisfy accountability needs. In this regard, 
the author considers vocational education as an intervention having consequences 
for consumers and society which need to be assessed through impact evaluation. 

The discussion will be limited to issues in assessing the impact of programs 
upon students and society. This 1 imitation excludes a variety of other 
possible appiications of impact evaluation, for example: 

1^ In investigating the impact of new methods of preparing vocational educators 
in teacher education programs 

2. In examining the impact cf federal monies on the existing vocational edu- 
cation establishment in a state 

3. In exploring the impact of selected efforts in vccational education 
research and development 

It should be noted that impact evaluation is not a specific methodology with 
a recipe to be followed. There are various methods for assessing the conse- 
quences of vocational education for student and society that can be used in 
impact evaluation. Therefore, the awareness of the nature and usefulness of 
impact evaluation should be increased by reviewing issues in evaluating voca- 
tional education. The bibliography that follows the text suggests items for 
further study. 



IMPACT EVALUATION: WHAT IS IT? 

1^ 

At this early stage students of evaluation theory and practice have not yet 
agreed upon a uniform definition for the term "evaluation," The rapid emer- 
gence of new developments in theory and methodology should deter premature 
closure on discussion. This makes it necessary to review a diversity of con- 
cepts in evaluation in order to trace the emergence of impact evaluation. 
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The CIPP Model 



Stiniulatetl by the mandate for program evaluation in the lilemcntary and 
Secondary liducation Act of 1965, the Phi nclta Kappa National Study Committee 
proposed a model for program evaluatioii using the following definition:^ 
"livaluation is the process of deUneatinM* obtaining and providing useful 
information for judging decision alternatives'' (Stufflcboam ct al-. , 1971, 
p, xxvj . Four types of evaluation wert- identified to correspond to four 
different types of decision-making needs. They arc as follows: 

K Context evaluation: defines the environment and diagnoses existing prob= 
ieiTiirr thcTrcby ideTTtifyLng unmet needs and unused Q])portuni t ics , in order 
to facilitate [planning decisions 

1, Input evaluation: identifies and assesses capabilities and the merits of 
varTouT st'r:itegies and iinplementat i on designs for forming decisions on 
the optimal utllizatioii of resources for achieving goals 

3. Process evaluation: reviews the operations of programs to generate feed- 
hriB;nnl^rmiiTi^^ (1) detect defects in design or implementation, 

(2) provide information for midstream decisions, and (3) accumulate 
documentation 

4, Produc t evaluation : measures and interprets attainments at the end of a 
.project cycle" (and during the project term as needed) to assess achieve- 
ment of goals 

The CIPP model (Context, Uiput , Process, Product) is well-suited for use by 
managers and admini st rators for planning, organizing, directing, and control- 
ling ongoing programs. 

However, dictionary definitions of Impact relate the term to the power to pro- 
duce a change In condition. Hvaluat ing the impact of vocational education 
|)rograms requires a clear conceptLon of the program as an intervention which 
causes desirable changes among its consumers by satisfying needs, addressing 
unmet needs, and alleviating problems in society. 



Drewcs and others (1975J attempted to adapt the CIPP concepts to vocational 
education but found it necessary to expand the scheme to include a separate 
"impact" component. Their purpose was to classify typical questions asked in 
vocational education. The following six categories of information were suggested: 

1. The context In which vocational education occurs 

2, The inputs into vocational education 

The process by wliich vocational education staff and facilities are 
structured and organized into programs to provide services 
'^1^^ product or output of vocational education programs, that is, program 
completions and withdrawals 
5. The i mpact or result of vocational education, such as postprogram place- 
merit and sat i s fact i on 

11ie pairwis e interrelationships between context, input, process, product, 
and impact elements 

= 2^ 9 , 



ThiH rupresunts a beginning Htop. although more work is ncodcd to clarLfy how 
various types of evaluation activitiuH couUl bo organized to produce the ro= 
qui rod information* Tho^c engagod in future work need to conHidor a larger sot 
of quustions and information. I'or instance, the 'Impact" category was limited 
to n few itcmH on postschool omploymont, employer satisfaction, and the satis= 
faction of alumni with their school programs and postschool jobs. This implios 
that (|uestions on inipact can bo answered from the information that is already 
available from follow-up surveys. However, such surveys arc not designed to 
investigate both intended and unintended consoquonces of programs, nor arc they 
easily adapted to studies of how results occur or fail to occur. 



Summative Evaluation 

Scriven (1967) uses the terms '^formative" and "summative" to distinguish between 
two types of evaluation activities with different purposes. In a curriculum 
research and development project, the activities comprising the formative 
evaluation are designed to improve the curriculum during its development. 
Those. of the summative evaluation arc designed to determine whether the finished 
curriculum package "represents a sufficiently significant advance on the avail- 
able alternatives to justify the expense of adoption by a school system" 
(p. 42). ' 

Actiiivlly the terms "formative" and "summative" have been used in a variety of 
ways. Stake (1967) perceives formative evaluation as oriented to curriculum 
developers, authors, and publishers; and summative evaluation as oriented to 
consumers/administrators, and teachers, Stufflobeam (1974) relates formative 
evaluation to decision making and summative evaluation to accountability. 
Bloom (1969) associates formative evaluation with providing "feedback and 
correctives at each stage of the teaching- learning process" (p. 48), to aid 
in learning; and views summative as baing used "at the end of a coursCj term, 
or educational program" (p. 47), to demonstrate achievement of mastery, or 
the attainment of instructional objectives. 

Scriven (1967) portrays the summativb evaluation of a new curriculum package as 
investigating the net advantage of this package compared to other alternatives. 
Similarly, studies on the impact of vocational education programs upon students 
and society should be concerned with the net advantage of the programs in 
question, but this requires explicit recognition of relevant alternatives. 

The issue was clarified by Kicvit at the 1978 National Conference on Outcome 
Measures for Vocational Hducation, Louisville, Kentucky. In discussing the 
personally and socially valued outcomes of vocational education. Dr. Kievit 
stated: "The issue is not whether those values might not have been attained 
by other means- = they probably would to some degree, but rather to demonstrate 
that vocational education is one means among others, equally effective in most 
ca^ und more effective than other means in some cases" (p^ 25). 

Tn fact, the problem in impact evaluation in vocational education is to produce 
credible evidence from at least one of several possible points of view. i..w 



mininium rcquiremGnt is to clearly attribute an obscrvgd outcome to the voca- 
tional program. Therefore , at least one comparison or control group must bo 
used. According to Campbell and Stanley (1963): 

Much roaoarch in education today conforms to a design in which a 
single group studied only once, subsecjuent to some* treat- 
ment presumed to cau^o change. such studies have such a total 
absence of control as to bo' of almost no scientific value,., se- 
curing Hcientific evidence involves making at least one compari- 
son, (p. 6) 

Whenever this criterion is not satisfied , discussing the findings in terms of 
impact of the program is not justified, lor cxtimplej much of the evidence on 
job placement rates is open to criticism because of failure to consider 
whether the persons might have obtained the same jobs without any vocational 
program, \ 

Moreover, the credibility of the evidence rests upon the comparison that is 
employed; for example^ bofore-and-af ter or trcatment-versus-control groups 
(Campboil and Stanley), and upon the care and precision with which it is 
carried out. For example, were the vocational graduates more highly motivated 
than the control group? At the high school level, simple comparisons could bo 
drawn between vocational alumni and persons from the general and/or college 
preparatory curricula; more complex comparisons would also control for differ- 
onces in ability, motivation, and other factors. 

More credible evidence from increasingly demanding perspectives would require 
assessing the impact of the intervention programs versus altamative programs ; 
here the evidence is comparative. This level of comparison differs from the^ 
earlier one, which implicitly compares results between those who did participat 
in a specific vocational program and those who did not. The more demanding 
type of comparison involves groups who have participated in different types of 
''programs," For example, comparative evidence about high school level voca- 
tional education could involve comparison between vocational alumni and non- 
vocational graduates who participated in (1) on-the-job training, (2) manpower 
and youth employment training, (3) postsonondary and proprietary schools, and 
(4) training conducted in military service. 

Researchers must account for the possible influence of other factors, such as 
different populations- and differential motivation. Nevertheless, credible find 
ings of this typo can be useful for decisions about the allocation of resources 
among competing programs. The decisions Vould also require comparative in- 
formation on costs, on the potential for service to special populations, and 
on other outcomes of the programs being compared. Ideally, cost-benefit studie 
would . be usefur for comparing competing programs on the basis of their pay-offs 
RiyUn '(1971) discusses the actual Usefulness of cost-benefit studies. 

Assessment of worth or merrt is an essential factor in evaluation. Popham 
(1974) has stated that ''systematic educational evaluation consists of a 
formal assessment of the worth of educational phenomena" (p. 8), At the same 



u 



time, the assoftsment of merit or worth of a prog rani should not hu mtorprctud 
an Lu\ aHseHsmunt of the merit or worth of 'its goals, or of the staff of the 
projn'am. 

Evaluating Effectiveness 
and Efficiency 

The analyHis of the cf fecti vcncsH of a program is an investigation of tho extent 
to which predGtorminod and dCHirod rcsultH arc produced. Accordingly, effec- 
tivenoHs HtudicB should have common fcaturcH with impact evaluation. However, 
impact evaluation should be concorncd with intended and unintended outcomes, 
including desirable or luulcsirablc results. 

The term ^'ef f iciency^* encompasses fiiore than ef fecti veness , as it designates 
attainment of desired outcomes with minimum expenditures of resources. It is 
not meaningful to describe a program as being both efficient and inoffoctive. 
The inciusion of both effectiveness and effort in the term ''cost-effectiveness" 
make It synonymous with efficiency. However, impact evaluation does not, by 
itself, involve consideration of resource levels. 



Pay-off Evaluation 

Scriven (19671 also proposes a distinction between '-intrinsic" and "pay-off" 
evaluation. Intrinsic evaluation of a program involves, an appraisal of the 
goals, content, and procedures of the program. An intrinsic evaluation of a voca 
tional education program would involve a review of objectives, methods, mate- 
rials, staff, and facilities, Therefore, intrinsic evaluation requires the 
formulation of standards for methods, materials, and other components which 
can be conceived as intermediate evaluation criteria. The intermediate criteria 
for intrinsic evaluation are very important, and the ralationships between 
satisfying intermediate criteria and achievihg overall goals must be clearly 
spec i f i ed . 

In spite of the weaknesses, intrinsic evaluation remains valuable because of the 
importance placed on intermediate criteria. If a program review can demonstrate 
^!;at many aspects of the program are considered to be successful, that the 
program seems to be personally satisfying to students, parents, and teachers, 
and that craft committees and other visiting review groups are favorably 
impressed, then this is meaningful evidence about the program. Indeed, 
Jencks and his colleagues (1972), claiming that education does not and cannot 
produce economic or social equality, suggest that schools should be judged 
on diverse standards, and especially on the quality of life that pupils and 
teachers experience in them. Although intrinsic evaluation is worthwhile, 
it is incomplete as a strategy for impact evaluation. 

Pay-off evaluation examines criteria by assessing the program through its 
effects upon students, for cxamplu, cm])loyment outcomes. Scriven (I9(i7) hu>»- 
gcsts that proponents of pay-off evaluation "would support their approach by 
arguing that all that really counts are the effects of the course on the 
pupils" Cp. 54), 

-5- 
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Since pay-off evaluation seeins conceptually similar to impact evaluation, it is 
important to note three. limitations of pay-off evaluation, 

1. In a multigpal program such as vocational education, it is ^difficult to 
formulate the complete set of evaluation criteria that an adequate pay-off 
assessment would require. Evaluators claiming to assess pay-off may be 
criticised for failure to address important outcome domains, 

2. In education evaluation it is frequently difficult to identify defensible 
operational criterion measures in each of the important outcome domains, 
for example, the pfoblem of formulating an adequate measure for assess- 
ing whether students have gained a ''better understanding" of curricUlum-^ 
materials, ' 

3. By itself, pay-off evaUifitiQn fails as Scriven 11967) suggests "to distin- 
,guish between importantly different explanations of success or failure" 

(p, 60). ^ 

William Foote Whyte (1972) stated that: 

It does us no practical good to know that a given training 
program reached a certain objective unless we can also discover 
what features of that program contributed toward that objective. 
Only as we make these observational studies, can we" have some 
confidence in being able to reproduce a program that has proven 
effective, (p, 284) ; 

Similarly, .-.ost-ben^it studies, which comprise a type of pay-off evaluation, 
are criticized because the cost-benefit ratio tells us little about how a 
successful program can be replicated, or how a faltering program might be im- 
proved, \ . 

Impact Evaluation 
Versus Research 

Evaluation* is designed to serve decision making for program improvement. This 
emphasis is said to distinguish evaluation from research CWorthen and Sanders, 
1973), Bvaluators are motivated primarily to emphasize worth or merit of pro- 
gramsi curricula, and materials, and decision making about educational policy 
and practice, Researchers try to produce conclusions that can be generalized 
in the process of discovering truth. 

Although the distinction, between evaluation and research may seem clear, clarity 
is lost in applying the distirfction to increasingly complex terms in tar litera- 
ture: "evaluation research," "policy research," and "social action research," 
These terms are also used to describe analyses of the effectiveness and, more 
broadly, the intended and unintended outcomes of policy and programs. It may 
be said that these terms are "broader" than program evaluations due to their 
emphasis on attempts to discover what elements of programs, and/or what kinds 
of programs, work for what groups, while the scope of program evaluation is 

' • ■ ,-,3 



thought to be limited to the analysis of designated programs. However, impact 
.evaluation should be concerned with what works, for whom, and under what condi- 
tions, and all the possible outcomes. 

The distinction between evaluation and research on the basis of methodology 
is also unclear. Stufflebeam and associates (1971) argued that available 
research methodology in the form of experimental design is not only inadequate 
for evaluation purposes, but in many respects is inappropriate.. Debates over 
appropriate evaluation methodology have, continued over two decades, as the 
following examples Lllustrate: 

Fstimatiiig the impact of education upon educational outcomes;; Coleman et 
al (19663; Bowles and Levin (19683; Smith (19683; Coleman (19683; Cain 
and Watts (1968 and 19703; Coleman (19703; Harvard Educ ational Review (1969); 
Leucke and McGinn (19753; Mosteller and Moynihan (1972); Spady (1973); 
Herriott and Muse ( '373); and Pedha'zur (1975) 

Estimating the impact of Head Start and follow-through: Cicirelli et al. 
(I960)' Rossi and Wright (1977); Datta (1976), and references therein; 
House et al. (1978); Anderson et al. (1978); Wisler et al. (1978); and 
Hodges (19783 

Estimating the impact of education on social and economic outcomes: Jencks 
et al. (1972); Berg (1971) ; Eckaus (1973) : Harvard Educational Review (1973); 
and Thurow (19753 _ ' 

Where new programs are concerned, some authors continue to call for modified 
experimental design (Meigs, 1975; Rivlin and Timpftne. 1975), while others remain 
skeptical about the application of the scientific method in evaluating human 
service programs. Still bthers are striking out in new directions: Aptitude 
and Treatment models (Cronbac.h et al.. 1976), Bayesian methods (Fennessey, 1976) 
Production models (Barnow, 19763, and Ethnomethodology (Patton, 197S) . 

Definitions of Impact Evaluation 
In Vocational Education , 

The elements of a working definition of impact evaluation in vocational educa- 
tion are as follows: 

1. Conception of vocational education as an intervention stresses the need to 
assess the consequences of programs upon students and society. 

2. Inclusion of intended and unintended results, desirable and otherwise, 
lasizes the wide range of consequences to be assessed. 



3, Assignment of results clearly due to the programs requires the use of yalid 
scientific methodology. 

4. Recognition of how results occur or fall to occur is necessary to guide 
decisions on both successful programs and those needing improvement. 
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Therefore, a working definition of impact cvaluatioh can be stated' 

Impact* evaluation in vocational education concoives yocational 
education to be an intervention ^ having consoquences for consumers 
and society which are to be assessed, by producing information on 
intended and unintended outcomes attributable to the program, in- 
cluding both desirable and other results, and illuminating how 
these Occur, for the purpose of satisfying decision making and 
accountability needs. ; - . 

IMPACT EVALUATION: 
WHY IS IT IMPORTANT? 

■? 

Vocational educators conduct impact evaluation in order to (1) improve voca- 
tional education programs, (2) satisfy accountability needs, and (3) fulfill 
the requirements of legislation. The first two reasons are legitimate purposos 
for impact evaluation. 

^'Improving vocational programs'' can include expansion of successful programs, 
revision of programs, .and termination of programs* 

Findings from impact evaluation can satisfy accountability needs by informing 
various audiences about the effects of vocational education programs. 

The third reason, legislative requirements, reflects the need for program im- 
provement and accQuntability, This perspective depends on the interpretation 
of Congressional intent and the classification of prescriptive legislative 
provisions 'according to the working definition of impact evaluation in voca- 
tional education. Three major areas,, data collection and reporting, planning, 
and evaluation, illustrate the concerns of the legislation. 

Data Collection and Reporting 

The Vocational Education Act of 1963 was considered a landmark piece of legisla- 
tion becausa it increased funding and broadened the definitions of federally re^ 
Imbursable programs. The provisions of the act resulted from a report of the 
the Panel of Consultants on Vocational Education appointed by the President in 
.1961, The panel reported that it was hampered in its deliberations by incom- 
plete, noncomparable, and inadequate data concerning vocational eduqation; for 
example, enrollments, completions, placements, and expenditures. The panel made 
the following recommendations for improvement: . 

To develop a statistical data reporting system covering enroll- 
ments and outcomes and indicating the need for continuing exist- 
ing programs and for developing new programs, and to assist the 
states in developing criteria to evaluate programs, to provide 
the basis for improving programs and to gauge the progress of ex- 
panding and changing programs,^ (Department of Health, Education 
and Welfare, 1963, pp. 25()-251) 

.is 



Thu Vocational liducatlon Act of 1963 also required the establishment of an 
Advisory Council on Vocational tjducatlon in 1966 to evaluate the results of 
the act, and to report, with reclmmendations for improvement , by January 1968, 
However, Evans, Mangum, and Pragan (1969) reported that: 

As the first advisory council on vocatioiMl education^ we have 
found it impossible to determine to our full satisfaction what has 
occurred under the Act ... .Despite the long, foreknowledge of the 
1966-67 assignmtint, no significant studies were undertaken with^ 
adequate leadtime to produce data for the council's needs, (p. p) 

Instead, the council found. "gaps in statistical data, deficiency in depth of 
reporting* as well as the lack of adequate standards for evaluation of per- 
formance^* (p, 39). The council stated that primary responsibility for these 
problems rests" at the federal level and with the leadership of vocational edu- 
cation* , ^ 

Stimulated by the 1963 and 1968 legislation, many states assumed responsibility 
for improving their reporting practices. In most, states, new or revised manage- 
ment information systems were developed CStarr et al., Geigle, 1978). 
The incomplete resolution of reporting problems, howevt iS apparent in 
hearings prior to the 1976 education amendments. For in^.^nce, Arthur Lee, 
Director of Project Baseline, testified that the lack of adequate data, in part, 
was the, failure of the U.S, Office of Education to require complete and proper 
data from the states, as well as a lack of uniformity of definitions underly- 
ing the data being repprted (Committee on Education and Labor, 1976), These 
concerns led to the provision in the 1976 amendments requiring the National 
Center for Education Statistics to develop and * implement the Vocational Educa- 
tion Data System, VEDS, The implementation of VEDS should provide further im- 
provement in ^reporting practices and result In more adequate data on enrollments, 
completions, and placements. 

Unfortunatel/, VEDS has nothing in common with "impact evaluation." The com- 
ponent in VTIDS related to , measuring impact is the collection and reporting of 
placement data. Data on the extent of employment with training-related occu- 
pations provide at best very^weak, and at worst misleading, information on 
impact, because of th^ambiguity of interpretation. Indeed, any placement data 
for a single specialty program should be interpreted in the context of such 
factors as: ^ 

1. The timing of the follow-up upon which the data are based 

2. The method of data col lection, that is from the graduates directly, or from 
teachers or counselors i 

3. The definition of training-relatedness 

4. The economic conditions , facing the graduates, and their responses to em- 
ployment opportunities by means of changing fields or relocating 
geographically 

' In addition to these factors in programs for entry level skills, the graduates 
may have been enrolled for diverse purposes, including avocational reasons. 
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These and other Gonsiderations place limitations on the interpretation of 
placement reports as information on impact. 



Planning 

In preparing for the 1976 amendments, the Committee on Education and Labor was 
distressed by reports from .the Government Accounting Office CComptroller 
General, 1974) and from the Center for Occupational Education, North Carolina 
State University (Drewes and Katz, 1975), that planning in most states was not 
coordinated among the several agencies providing education and training^ and 
that the existing planning mechanisins were not based upon data concerning 
projected manpower needsV As a result,^ 1976 amendments contain pre- 
scriptive provisions for the coordinating of planning and use of manpower data. 
However, reviewing manpower data suggests the data is unrelated to the impact 
of vocational education; - . 

The manpower requirements approach to planning vocational education programs 
finds its rationale within the legislative mandate to provide training that is 
realistic In terms ofoopportunities for gainful employment . Methodology for 
educational planning to meet manpower requirements, applied in the Mediterranean 
Regional Project of OECD (Parnes, 1964) ^ has been adapted to identify the 
relevant employment opportunities* 

In the Mediterranean project, manpower planning was undertaken to avoid inadequate 
supplies of skilled manpower during economic development. Such an application 
presupposes national goals,^ such as desired rate of growth in GNP, from which 
industrial and occupational manpower /^requirements^' are derived and then com- 
pared to anticipated levels of manpower Supply or projected graduates; Esti- 
mated discrepancies between desired and expected manpower supplies are then 
considered for planning expansions of education and training systems. 

The application of manpower planning to vocational education differs from the 
Mediterranean project in important ways. The motivation is to locate relevant 
job opportunities for persons assumed to need. some type of training and thereby 
discover the types of training that should be offered. Economists have 
challenged this assumption on throe points: . ^ 

1 Assuming that persons will need some training, it may still not be 

necessary to plan in advance to offer specific programs except for occu- 
pations involving levels of skill that require long periods of training, 

2. Assuming that it is possible, to identify the specific types of train- 
ing that will be needed, it may not be necessary to plan publicly-supported 
programs. Adjustments in the market encourage workers to obtain useful 
skills or employers to provide the needed training without public outlay, 

3. Granting the need for public support, public provision of the training may 
not be necessary. Student aid or direct subsidy to institutions can 
provide the needed training by regulated proprietary schools (Nolfi et al., 

^ 1977, chapters X and XI) • ^ 
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There are valid reasons for a limited role for manpower forecasts in planning 
vocational education programs. Young and others (1971) argue that simply 
knowing the occupations having largo numbers of openings is not all that is 
relevant to decision making. Many occupations such as clerical, allied health, 
and service occupations have high turnover characteristics because of abnormally 
low rates of pay. Students may also continue to enroll in vocational courses 
for other than vocational reasons, for example, automobile and airplane 
mechanics. Although the number of projected net openings should be considered 
in program planning, many other factors are also important, such as entry-level 
wages, wages at seniority, student interests and desire to relocate, job satis- 
faction of occupational incumbents, employer and job entry requirements, pro- 
gram costs, and the potential for service to special student populations. 
Moreover, this assumes that manpower projections are accurate (Stevens, 1976) . 
and can clearly be related to specific vocational programs. 

However, planning programs is clearly different than assessing the imijact of 
programs upon students and society. Therefore, the requirements for planning 
in the 1976 amendments should not be confused with the need for impact evalua- 
tion. 

Evaluation 

The third area of emphasis in the 1976 amendments consists of prescriptions for 
evaluation. The rationale for this. is found in the Committee on^ Education and ^ 
Labor report (1976, p. 20). As stated earlier, the legislative requirements for 
VEDS and Section 112(5) evaluations do not necessarily ' imply impact evaluation. 

However, other provisions of the legislation bear more directly on ^ha need 
arid opportunity for impact evaluation. First, Congress requires the state 
advisory councils to '^evaluate vocational education programs, services and 
activities assisted under this Act, and publish and distribute the results, 
thereof" [Sec. IpS (d)(2)i. Second, the states are required to submit annual 
planning and accountability reports to the Commissioner. Such reports must 
contain, in addition to the placement ,and employer data provided in Sec. 112 
(b) (1) (B), more stringent evaluation under Sec, 112(b)(1)(A), namely: 

Each State shal 1 ... evaluate the 4f|^^ctive of each program 
within the State being assisted with funds available under this 
Act; and the results of those uvaluutions shall be used to revise 
the Staters programs, and shall bo made readily^avaiiable to the 
State Advisory Council. 

This section is interpreted in the Federal Register as including: 

(a) Planning and operational procosses, siich as 

(1) Quality and' avallabi 1 i ty of instructional offerings 

(2) Guidance, counseling, placement, and follow-up scrvicos 

(3) Capacity and condition of facilities and equipment 
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(4) Employer participation in cooperative vocational programs 

(5) Teacher/pupil Tatios 

(6) Teacher qualifications . 

(b) Results of student achievement, as measured by standard 
occupational proficiency measures or other methods 

(c) Results of student employment success, such as wages, 
employment and unemployment, and employer satisfaction 

(d) Results of additional services, including service to special 
populations (October 3, 1977; Section 104,402) 

All of the foregoing imply impact evaluation. 

Third, the states^ annual applications will: • • . . 

(a) Describe the /ocational education needs of potential students 
in the area or community and indicate how, and to what 
extent, the program proposed in the application will meet 
such needs, and 

(b) Describe how the findings^of any evaluation of programs" 
operated by such applicant during previous years... have 
been used to develop the program proposed in the application, 
[Sec, 106 Ca) (4) CB)] . . 

These evaluation requirements imply the need for evidence on impact that is 
not ambiguous and is of adequate quality for planning. The law does not specify 
or Refine 'impact evaluation/V or require the states to use selected statistical 
methodology or specific sampling techniques.^.. The emphasis is to require the 
type of evaluation that produces valid and reliable findings for decision 
making and accountability. . . 

The Congress has promoted and supported vocational education since the Smith- 
Hughes Act of 1917, In writing about the implications of the latest act, 
Stdvenson (1977) declares the Education Amendments of 1976 to be the most * 
prescriptive provisions in history, and concludes that "Congress has assumed 
responsibility for molding thtf shape of vocational education for the future'^ 
(p. 2). However, even the Smith-Hughes Act of 1917 specified that: 

, To participate In the benefits of the legislation, the States 
were required to... prepare a State plan showing the programs 
they intend to provide (and) make an annuab report to the 
Federal Board for Vocational Education showing the work done 
during the year, (Department of Health Education and Welfare, 
Panel of Consultants, 1963, p. 22) 

Thus,' the precedents for Congressional promotion of data collection and report- 
Ing/planning, and eva luat ion arc more than fifty years old, mule some per- 
sons may conceive the pre^^ont formulation to bo a burden, others view it us a 
logical consoquunce ot; nn evolutionary process covering more than a half-cen- 
tHVy of federiil involvement. 



CURRENT EVALUATION PRACTICES 



It is necessary to review qxisting evaluation practices in vocational education 
in order to discuss their relationship to impact eyaXuation, According to our 
working definition of impact evaluation in vocational education, its purpose is 
to satisfy decision making and accountability needs. States that perfoiTn impact 
evaluations, that use findings for program planning and improvement, and that 
publish documentation of the evaluations and their use are being responsive not 
only to the provisions of the 1976 education amendments but to their clients 
and to the general public. 

It i-s important to understand the strengths and weaknesses of the evaluation 
models that are now in use. .Ambiguous evidence from weak evaVuiition designs is 
not suitable either for facilitating decision making or for si^tisxyinj^ accounta- 
bility noeds. Moreover, the amount of funds available for restiarch and evalua- 
tion is small in comparison with the many competing nc^eds in oducatian. Pro- 
posals for funds for impact evaluation should be carefully evaluated for their 
potential for building a strong,^, long-term compatibility for impact evaluation 
in vocational education, 

"Evaluation'* Reporting 

Vocational education exists for providing education arid training to students, 
and the nuinber of students that are served constitutes an Indication of the 
success of programs. Indeed, the 1976 amendments envision that: 

Persons of all ages in all communities of the State, those in 
high school, those who have completed or discontinued their iurmal 
education and are preparing to enter the labor market, those who 
have already entered the labor market, but need to upgrade their 
skills or learn new ones, those with special educational handicaps, 
and those in postsecondary schools, will have ready access to 
vocational training or retraining. (Sec. 101) 

Data are needed to show service to special popur^.tions , Regardless of the need 
for information on the extent of service and of the benefits of improved 
management and reporting systems, the enumeration of students does not 
constitute evidence of. the impact of programs. The size of current or pre- 
vious enrollments bears no necessary reflation to program quality or effec- 
tiveness. 

Indeed, the management information systems being developed to improve reporting 
practices may fail to contribute, directly or indirectly,, to impact evaluation. 
In a Rand Corporation Research Study, Coleman and Karweit (1972) set out to 
study administrative data systems in education 'Vith the aim of designing 
cor» puteri^ed systems which would facIiLtatc such administration, and also pro- 
vide data for research and evaluation purposes^' (p, ix) , They were especially 
interested in Information "about the exposure of students to possibly effective 
educational environments" (p, 6). However, they discovered that "systems must 
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be appropriately designed from the outset to bring the power of the infdrmation 
to bear on education decisions^^ (p, 4), Otherwise^ that data might be unsuitable. 



Even when carefully designed, information systems may fail to produce the 
anticipated benefits, Lucas (1975) studied many information systems that 
'*would have to be classified as failures. If a system is not used, it cannot 
be considered a success even if it functions well technically'* (p, 3), 

In order for more information systems to be useful in impact evaluation, they 
must contain the type of systematical lyrcollected and woll^organized informa- 
tion that can form the basis for longitudinal files of the progrcsH of ouch 
student through a program* Questions fur which such Lnforiiiat ' on may be useful 
include: 

1, How F -ny students are enrolled in the program of their first choice? 
Sec Olid choice? 

2, What are the educational goals of current students? 

3, Within eaoh program, how many students are definitely planning to pursue 
careers related to their programs? How many students subsequently change? 

4, What is the basis for their enrollment in the program? Of those who report 
career interests, what is. the extent of their knowledge of career opportuni- 
ties? What is the validity and reliability of the caraer information they 
possess? . : 

Research in Pennsylvania has verified the feasioility of collecting and using 
such information in assessing the impact of programs (Impellitteri and KapeSj 
1971; Kapes et al,, 1974]. Kapes and associates (1974) have reported that the 
information proved useful in addressing the rjncern that many outcome studies 
focus on postschool experience while failing: 

To provide infoi^matlon regarding the nature of the inputs to the 
vocational education program [the ,type of youngster who typically 
selects such programs) so that valid Inferences regarding effects 
of the program can bo made, (p, ()) 

In Fall of 1963, 100 percent samples of ninth-graders in public high schools were 
selected within three large school districts* Longitudinal files were accumu- 
lated with follow-up data during and after high school, The data files con- 
tain scoros on a variety of demographic variables, test batteries, interest 
inventories, and performance measures* The results have been used in more than 
20 completed studies on such topics as the stability of occupational goals 
(0*Roilly, 1973)* llerr et al* (1975) have also used the data to show the rela- 
tionship between high school experiences and postschool success and satisfac- 
tion. Special reports have also boon written for the participating districts* 
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A HubstantLul amount of work on vocar iuiia T education han bicn completed with 
the large' surveys (Grasso and Shea, forthcoming a and b) because of the ready 
availability of background and other required information. 

In local education agencies, student record files and school record files may 
contain relevant data. The absence of systematically collected comprehensive 
information on vQcational students, and on the larger population from which 
they were drawn, may discourage impact evaluation. In such a situation impact 
would be difficult to analyze because of the inabilit/ to separate program 
effects from preexisting differences* 

Therefore, it is also important to understand the process by which students 
choose a vocational program, or are tracked into curricula, or change curricula 
during enrollment. Searches of documents in ERIC and NTIS have uncovered 
little work on these topics. Yet their importance is evident in the following 
examples of existing evaluation studies: 

1.. A haphaEard system of intake may have negative consequences in terms of 
rates of "noncompletion J* Consider this from a study of vocational 
education: 

This points to one of the basic flaws in the provision of voca- 
tional education in New York State outside the large cities, The 
control of the flow of students,,. is totally beyond BOCES (Board 
of Cooperative Education Services) What coordination and artic- 
ulation there is between BOCES offerings and sending school of- 
ferings is voluntary and at the discretion of the sending schools. 
What contract there is for recruiting students is at the dis- 
cretion of the sending school ... .This has resulted in uninformed 
parents, students and teachers, overlap of courses, violently 
fluctuating enrollments in vocational courses, and an inability 
by BOCES to predict and plan, (Swanson, 1976, p, SI) 

2, Studies based on student data ranging from national survey data CGrasso 
and Shea, forthcoming a and b; Jencks et al,, 1972) to case studies 
(Roscnbaum, 1976) show that relatively high proportions of noncollege 
preparatory students are desiring to go to college. According to one 
author, a reason is that "the cholcoH made by parents and students are 
not likely to be free and informod choices" (Rosenbaum, 1976, p* 124)* 
Such points would appear to have implications for several traditional 
outcome measures of "success,** 

3. The extent to which students change from one curriculun\ to another will 
affect findings on the effects of curriculum by undermining the validity 
of the curriculum measure. For instance, if the general program does 
not serve a student well and the student transfers belatedly to a voca- 
tional program, then a follow-up of this 'Vocational* graduate may not 
only conceal the shortcomings of the general program, but fail to reflect 
the reputed benefits of participation in a complete vocational "program. 
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4, The timing of curriculum choice varies among districts and states, 
and in some casesi within districts. The failure to recognize this 
in studies of the impact of vocational education on the dropout rate 
can lead to fallacious conclusions. An exemplary study based on 
adequate student record data would attempt to conceptualise two 
distinct "effects": the power' of vocational programs to attract 
youth who are considered to be likely to drop out and the holding 
power of the curriculum. An implication is that studies that show 
results favorable to vocational education may be viewed skeptically 
by researchers because of the treatment of cases of youth who left 
school prior to the time when enrollment in a vocational program was 
permitted. They frequently are classified as dropouts from the 
general curriculum, but may also be classified as those, whom the 
vocational program failed to attract . 

These examples suggest the need for comprehensive student and school records in 
order to permit the application of adequate evaluation designs. 

Evaluation by Site Visitation 

The on-site visitation is a widely accepted method of assessing program quality 
which, in several respects, resembles intrinsic or process evaluation. The 
on-site team examines materials, curriculum, equipment, and techniques on the 
basis of standards that may be regarded as intermediate evaluation critaria, or 
attempts to observe whether the program is conducted in agreement with its 
goals and objectives. The Amefican Vocational Association has developed an 
evaluation modil patterned after educational accreditation studies (Ash, 1973) 
that consists of many intrinsic and process evaluation components* Most states 
employ this type of evaluation. 

Comprehensive evaluation systems would include . both on-site and impact com- 
ponents. However, on-site visitations do not constitute impact evaluation. 

Evaluations of Student Competencies 

Student competence is one factor In impact evaluation whicli determines the 
consequences of the program for the student, Criteria consist of all the 
proximate effects of the program, including cognitive, psychomotor, and affec- 
tive outcomes (Wentling and Lawson, 1975, chapter 3), These proximate effects, 
such as occupational skilU, constitute the means by which the student may 
achieve long-range goals* 

Programs for entry level job skills should be evaluated on the knowledge and 
skills that are the minimum behavioral object^Dves of the instructional program. 
Criterion measures may be the results of locally designed or standardized 
written tests and performance tests (Erickson and Wentling, 1976)* At the local 
level, findings may be compared with minimum standards to determine the propor- 
. tlon of students achieving mastery. At district or state levels, variations in 
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program sLte^i offer the opportunity for statistical unulysiH that could demon- 
strate the effects of various program components upon student competencies, 
such as duration and intensity of program, combinations of classroom and ex- 
perimental teaching, and use of materials and equipment. Such analyses should 
be verified by pilot experimentation* Extensive revisions on the basis of cor- 
relational data may be unwise. 

Instruments for assessing job knowledge and generally useful labor market skills 
from the National Assessment of Educational Progress CNAEP) can be adapted for 
impact evaluation (Ahmann, 1977). NAEP data have already proven useful in 
research on race and sex occupational stereotyping CGottfredson, 1978). Also^ 
instruments for assessment of career education programs might be adaptable for 
measuring labor market knowledge and job^-finding skills. 

Types of knowledge and skills other than in the specialty area comprise im- 
portant criterion measures, including basic skills for secondary- level voca- 
tional students, Parnell (1971) asserts that vocational programs contribute 
positively to general academic goals: 

Relating formal, planned instruction to the life«career roles 
of students will help bring about a marriage between ^'academic** 
and '^vocational" programs and will help students find a new 
significmce in learnings thus motivating them as so-called 
academic courses never could do. (p, 102) 

However, research suggests that a prime determinant of achievement is students' 
time spent "on task." From thisj one might hypothesize that the lesser amount 
of time spent by vocational students in general academic subjects would have 
negative consequences in terms of basic skills. Moreover, the need in voca- 
tional education programs to serve populations who are handicapped in terms of 
their entry level basic skills makes it important to understand the effective- 
ness of the program in developing basic skills. 

Other benefits of vocational education, such as enhanced self-esteem^ may be 
Incorporated into impact evaluations of proximate outcomes. Relevant measures 
are available from existing work in the social sciences. Constructing and 
using now or unusual measures is sometimes risky when findings and interpreta-' 
tions arc reviewed. For example, Egginton C1978b) wrote: 

- Despite the massive investment of funds in vocational education 
programs and despite the recognition that vocational education 
curricula should address the problem of students' negative values 
and attitudes, the vocational education student still holds him- 
self in low esteem and tends to treat learning with disdain, 
(p. 534) 

Further examination reveals that the measure of negative self- image/dofensive- 
ness was: 
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.Do you usually feel like the world is against you? 



1, Yes 

2. No 

This measure has little to recommend its use. It is remarkable that expansive 
interpretation was based on a small difference iri the proportions responding 
"Yes*- (23,0 versus 21.2 percent), hurthermore, no measure of preexisting dif- 
ferences in so-called self-image wan utilized in this study (Egginton, 1978a). 

Follow-Up Evaluations 

Information on the experiences of program completers and leavers provides 
promising opportunities for impact evaluation. Performance of graduates after 
leaving the program generates criterion measures that indicate the enhancement 
of the lives of participants. ' 

Follow-up studies on program completers and leavers have been conducted in most 
states. Often the proceduriss are limited in length and frequency of follow-ups, 
the population coverage, and the substantive inquiry. Low rates of response often 
preclude confident interpretation of findings (Ball and Anderson, 1975; Headrick^ 
1977; Lee and Sartin, 1973; Paul, 1975 and 1976; Sparks, 1977), Even so, voca- 
tional education compares favorably with 'many other types and systems of educa- 
tion in the extGRt^to vvb4H;h postprogram experiences of graduates are examined 
at all. ""^^ 

At state or substate levels, follow-up studies of alumni and employers are con- 
ducted for several types gf impact anulyses. Regular follow-up surveys seek to 
determine the postprogram status of alumni in a number of dimensions. In one 
case, an Illinois follow-up for high school programs (Felstehausan et al,, 
1973) obtained information on: 

1. Influences on occupational decisions 

2. Posthigh school employment and school enrollment status 

3. Alumni assessment of program 

4, Employer assessment of alumni work preparation ^ ■ 

5, Employer assessment of Important qualities and skills of prospective 
employees 

6, Extent of training^rolatod employment 

Data collocted on those six factors did not indicate that the program was sue- 
c^essful, although graduates were likely to rate their programs highly . The 
methodology used did not ^Uqvv the researchers to draw conclusions about the 
apparent discrepancy in those findings. Analysis of data from the National 
Longitudinal Surveys (GrasHo and Shea, forthcoming a and b) suggests the im- 
portance of a clear concoptlon of the attitude being measured when students 
rate their programs highly: 

Evidence suggosts that occupational students do not like high 
school as much as other students, but that both occupational and 
other students tend to like very much the vocational courses that 
they have taken. 
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The same care is required for use of psychological measures. Findings on 
self-esteem may differ from those concerning what might be called uca^lemic 
self-esteem.. 

Comparative data on the experiences of nonvocational students are also required 
LO better understand the tracking, selection, and socialization procGSses that 
operate within the secondary system. In Minnesota, a limited .survey has been 
field- tested for collection of information on all high school students [Copa 
et al,, 1976; Irvin and Copa, 1974). The State of Florida practices follow-up 
evaluation for all secondary school leavers; however, the potential for utiliz- 
ing this comparative data for impact evaluation in vocational education has 
evidently not been realized. 

Comparative data are especially important for studies of earnings of graduates 
and of the economic efficiency of programs CStromsdorfer , 1972). Many states 
have performed some type of cost-benefit study (Ghazalah, 1975; Kraft and 
Weisman, 1970; Paul, 1976; Sparks, 1977; Swanson, 1976), However, when com- 
parative data on the postschool experiences of nonparticipants are unavailable 
and estimated values are used instead, the findings may be challenged* 

Impact studies of special programs of various types in vocational education 
(Snmurs et al,. 1971; StromHdorfer and l^ackler, 1973;^ Walsh ot al., 1976a and b) 
and other programs (Development AssociateSi Inc, 1972; Perry Jt al . , 1976; 
Sprengle and Tomey, 1974; Stromsdorfer and Fackler, 1973; Moayed-Dadkhah, 
1975) offer exemplary study designs that take advantage of available information 
on background, preexisting differences, and outcomes among comparison groups 
of nonparticipants. Other interesting studies compare outcomes of alternative 
types of vocational programs; for example, several studies performed comparative 
analysis on vocational programs conducted by puBlic and private institutions 
(Wilms, 1974; Wdlman et al,, 1972), One longitudinal study on manpower 
training (Cooley et al., 1975) was facilitated with data on earnings obtained 
from the Social Security Administration, Another longitudinal study on a 
sample of youth from one state, was possible through the cooperation of the 
Social Security Administration and the State Tax Department which provided 
earnings data (Sewoll and Hauser, 1975), ^ 

Recently impact evaluation in vocational education has become more , sophisticatod , 
A cost-benefit study on cooperative education dealt with impact fr^m several 
distinct perspectives (Cohen et al . , 1977), The costs and benofits to the 
participant, the institution, and the employers were conceptualized and ana- 
lyzed separately. Moreover, the authors deal with a limitation of cost-benefit 
analysis by separate analysis of probable limitations to the potential expansion 
of cooperative education programs, A study of economic returns to secondary- 
levol occupational education conceptualized and measured various components 
of economic effects (Fisher et al,, 1976), Specifically, the programs were 
designed to have both general and specific training components which conferred 
earnings advantages directly, as well as indirectly through enhanced probability 
of completion of high school, longer work experience (that is, less time spent 
not working), and complenienting whatever postschool training may bo received, 
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In vocational follow-up studies conducted by states, the timing of the follow-up 
presents a problem because, as several authors (Conroy and Diamond, 1976r Kauf- 
man et al., 1969) suggest, initial earnings advantages disappear in a few years. 
The advantage may even be rfrversed over a longer period. Although alumni follow- 
ups conducted after a short period of time are advantageous in terms of rela- 
tively high rates of survey responses, the appropriate time reference in impact 
evaluation is the long-term assessment of the program impact over the working 
career. Unfortunately long-term follow-up data refers to graduates who completed 
their programs years ago, and findings reflect the effects of programs that op- 
erated at that time. These factors suggest the advisability of both short- and 
long-term follow-ups. 

Data from national longitudinal studies have been analyzed on topics such as 
wage advantages and wage progression over tijnc. Studies have used the National 
Longitudinal Surveys of Work Rxporience (Grasso, 1975; Grasso and Shea, forth- 
coming a and b) to examine the impact of vocational education and training. The 
data source refers to young men and women who attended high school during the 
1960s; the follow-up data refer to the early 1970s, A range of labor market cri- 
teria'were examined, including unemployment experiences, occupational assignment, 
hourly wage, annual earnings, and expressed satisfaction with jobs. 

Other studies have. used data from the National Longitudinal Study of the Class 
of 1972 (Creech, 1974; Creech et al . , 1977; Echtemacht, 1975; Freeberg and. 
Rock, 1975; Nolfi et al,, 1977) to investigate the effects of curriculum on 
measures ranging from self-esteem to wages and occupations. The availability 
within these national data files of Information on background, personal 
characteristics, education, and postschool labor market experience facilitates 
the application of comprehensive analytic designs. Studies using these files 
may suggest improvements in the design of impact evaluations, 

Grasso and Shea also investigated the question of occupational assignment by 
comparing the occupations held by secondary-level vocational graduates and 
those held by their nonvo-^ational peers; that is, those who did not go to 
college. The availability of information on occupation and industry in which 
each was employed permitted the use of a variety of measures of ^'skill require- 
ments** of the jobs hold. Overall, the results contained no clear evidence to 
sugrTcst that vocational graduates obtained jobs requiring higher skill levels 
than did nonvocational graduates. This suggests the need to incorporate infor- 
mation on suitable comparison groups in impact evaluation, and the need to use 
nuasurcs other than traditional measures, such as training-related placement. 
(For an illustration of such an attempt within vocational education, see Copa 
and Kleven, " 1977. ) 

Findings based on both of these national "data sets do not provide strong 
support for the proposition that vocational education represents a labor 
market advantage for males. For females, the vocational program is associated 
with higher pay than are the other curricula. For both males and females, 
participation in postsecondary education and training appears to confer labor 
market advantages. However, even hero, several considerations, sharply restrict 
the usefulness of findings and their interpretations for decision making, 
Including: 



Lack of information on' the nature of the '^program," its duration and 
intensity, and its costs 



2, The impossibility of identifying oxccptionally effective and ineffective 
programs 

There may also be substantial effects for participants that have not been 
measured. For example, positive benefits for participants may exist in the 
capacity to "do-it-yourself," Consumer and homemaking programs and automobile 
repair are examples of programs teaching skills that are potentially useful in 
this way. It may be possible to estimate the Impact of these programs. 

The focus on impact to this point has been directed to participants in voca- 
tional education, nonparticipants, and employers, A few studies have also 
attempted to measure impact on society, Bolino (1972) studied the impact of 
occupational, adult, and various nonformal education programs upon society. An 
earlier study on the contribution of regular formal education to economic growth 
Muring the 20th century indicated that education accounted for twelve per- 
cent of the gross national product between 1900 and 1929, and for twenty- 
three percent from 1929 to 19S7. Bolino' s results might be interpreted to 
suggest that, of the national economic growth occurring from 1900 to 1957, about 
five percent may have been due to occupational and adult education and other 
education and training outside of regular schooling. 

The literature review for this paper uncovered no studies evaluating the "impact 
upon society" at the state or local levels. The only pr^^.sible exception 
(Tuttlo and Alexander, 1976) considered the role of vocational education in 
local economic development in terms of attracting new industry and serving 
expanding itidustry. 

The analysis, based on experience in one state, is interesting although the 
relationship to impact is unclear. In its discussion of the concept of 
"attracting new industry," it fails to consider whether the geographic move- 
ment of firms provides new opportunities in the receiving area while simultane- 
ously reducing the opportunities in the sending area. Further analysis is 
needed in order to assess the impact of such cases on society. In the dis- 
cussion of service to expanding industries, the analysis does not consider 
the underlying process governing the expansion of service. For example, it 
is not clear why expanded service for expanding firms should be given higher 
priority than improved or expanded service for stable firms, 

The issue of the impact of vocational education on unemployment constitutes a 
similar case. Proponents may claim that the programs will combat high unemploy- 
me'nt rates, but since vocational education does not create jobs, it does not 
expand the employment base. Moreover, studies may show that during the period 
following high school graduntion, vuc-itional graduates iocutc their first jobs 
faster than other graduates, but whether this phenomenon is due to occupational 
skills, placement asHistanco» or other influences is not clear, Data from the 
National Longitudinal Surveys (NLS) over several years do not indicate any ad- 
vantage terms of employment for male vocational graduates. The overwhelming 
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influence on NLS unemployment data is the state of the oconomy (Grasso and 
Shea, forthcoming a and b) , Impact studies would need to reveal beneficial 
effects on unemployment rates and to demonstrate how these occur, in order to 
improve decision making on expanding vocational ' job-finding assistance programs. 

Critics of vocational education assumD that, in the absence of public support 
and provision of vocational education, employers and/private institutions would 
provide any required training* Since much of vocational education is directed 
at developing rather general skills that would be useful to a large number of 
potential employers s it is not clear how responsive firms would be to under- 
taking this training function. Private training schools may be willing to expand 
but extensive student aid programs would be required to offset the unequal 
access that students may have to funds. Moreover ^ the history of strong local 
support for vocational education suggests that a shift from public to private 
provision would need to occur gradually. Students and society may place more 
importance on the goals of vocational education rather than developing specific 
marketable skills. Therefore, public vocational education may be rational and 
more efficient than the alternatives for providing socially optimal levels of 
training. 

Interest Inglyj this logic does not involve discussion of wage or occupational 
advantages among vocational graduates. Instead, it suggests a socially 
optimal level of occupational training with the benefits captured partly by 
students, partly by employers, and certainly by soci ety-at-large. Impact eval- 
uation of vocational education in these terms has not yet been undertaken. 



CURRENT DEVELOPMENTS 



Evaluation Models for States 

In addition to requiring extensive evaluation^ the Hducation Amendments of 1976 
provide for a number of evaluation efforts that may prove to be helpful to 
states in conducting evaluation. 

Section ll2Ca)(2) requiros that the Bureau of Occupational and Adult Education 
review and analyze the strengths and weaknesses of the programs assisted under 
the act in at least ten states^ each fiscal year beginning October 1, 1977, and 
ending September 30, 1982, Consistent with these responsibilities and to assist 
the states to fulfill their responsibilities under Section 112 (b)Cl), the 
Bureau awarded a contract to CRC Education and Human Development , Inc., and 
its subcontractor Mary Ellis Associates, Inc, to assist the states by develop- 
ing and providing the rough framework of a model evaluation subsystem. The sub- 
system was to consist of three components: process evaluation, product/outcome 
evaluation, and product/impact evaluation. 

According to the plan, the proceHs component was to be developed with the 
benefit Sf existing procass models^ (Ash, 1973; Starr and Dief fenderf er , 1972) 
and would contain the Following categories: 
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1. What is being done in vocational education? 

(a) Program 

(b) Curriculum 

(c) Ancillary services 

2. Why is this being done? 

(a) Objectives ^--^ 

(b) Student needs ^ ^ 

(c) Employer needs . ■ ' 

3. ^Vho participates in delivering vocational education? 

(a) Administration 

(b) Instructional ^ ' d support staff 

(c) Employer participation 

4. Who is receiving benefits of vocational education? 
(a) Regular students 

7 (b) Special populations 

5. How and under what conditions is vocational education being delivered? 

(a) Administration and financial management 

(b) Facilities and equipment 

(c) Resources 

(d) Economic conditions 

The product/outcome component would include career development and labor market 
outcomes for completers and leavers with recommended follow-up procedures, in- 
cluding questionnaires for alumni and employers. For example^ the draft 
version of a questionnaire for former students included sections on educa- 
tional history and current status, employment history and current status ^ 
postCompletion or additional training, and job satisfaction and advancement. 

The product/ impact component would include "impact-' conceived as student com- 
petencies and measured by criterion- referenced tests and standard occupational 
proficiency tests. This component was to rely on existing instruments; no new 
instruments were to be developed. 

Unfortunately, in October , 1978, the contract was cancelled before a final review 
and- field-testing of the draft versions of the process and outcome components^ 
and before the development of the impact component* State and local staffs 
may find it useful to review the draft version of the outcome component > 
especially since it was developed by means of extensive review of current 
practices in follow-up evaluation. 

Vocational Education Study - " 

Section 523 (b) (1) provides that the National Institute of Education shall: 
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Undertake a thorough evaluation and study of vocational education 
programs, including such programs conducted by the states, and 
such programs conducted under the Vocational Education Act of 1963, 
and other related programs conducted under the Comprehensive 
Employment and Training Act of 1973 and by the State Post- 
Secondary Commissions authorized by the Education Amendments of 
1972, Such a study shall include-- 

(A) A study, of the distribution of vocational education funds 
in terms of services, occupations, target populations, 
enrollments, and educational and governmental levels and 
what such distribution should be in order to meet the 
greatest human resource needs for the next 10 ye^rs; 

(B) An examination of how to achieve compliance with, and 
enforcement of, the provisions of applicable laws of the 
United States; 

' CC) An analysis of the means of assessing program quality 
and effectiveness; 
(D) A review and evaluation of the effectiveness of (Consumer 
and Homemaking Programs), 

In early 1977, NIO commissioned a series of preplanning papers to assist in 
formulating a study plan CGrasso and Shea, forthcoming b) , A plan for the study 
and evaluation was sent to the CongrcsH on necember 30, 1977 by Henry Uavid, 
Study Director, and included studies on the following topics: 

A, Distribution of funds 

1. Quantitative Descriptive Study of the Distribution of Funds (Notei 
Awarded to University of California; Charles Benson, Principal 
Investigator) 

2. Case Studies, on Meeting Special Needs 

3. Projecting Human Resource Needs and Funding Levels and Distributions 
' 4, Simulation Model 

5. Evaluations of Outcomes 

B* Compliance ^ 

1. Inventory of State Legal, Administrative, and Fiscal Practices for 
Vocational Education; Case Studies in Fifteen States; and Analysis of 
Federal Legislative Framework (Awarded to Abt Associates, Inc.) 

2. Study of Federal Capabilities for Implementation 

3. Study of Compliance and Enforcement in Selected Federal Grant-in-Aid 
Programs 

4. Vocational Education-CETA Coordination 

C. Means of Assessing Program Quality and Effectiveness 

1. Review and Synthesis of Existing Information (Awarded to CRC Education 
and Human Development, Inc.) 

2. Conference on Evaluation (Conference on Effects of Vocational Education, 
June 1978) 

3. Case Studies In 10 States 

4. State Survey of Evaluation Practices 
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D. Consumer and Homemaking Education . , , ^ 

1 Tho Responsiveness of the C and HE System to Congressional Intent- 
Federal, State and Local (Awarded to„ CRC Education and Human Develop- 
. ment. Inc. ) 

2. The Effectiveness of C and HE ' ^„,,.„--* 

3. Projecting What People Need to>Qw in 1982 and Beyond to be Intelligent 
Consumers and Effective Homaflrgkers ci,-n= anH 

4. How People Now Acquire Consumer and Homemaking Information, Skiiis, ana 
Abilities .from Sources Other Than C and HE Programs 

Several of these itehs relate to impact evaluation. First is a review of 
research based on various national data bases that involve vocational education 
issues (Grasso and Shea, forthcoming b) . Second is a review of state practices 
concerning short-tepm follow=up studies (item C-1 above). A third review of 
any other recent evaluation research that relates to economic, social, and 
educational outcomes is yet to begin Citem A-5 above). Finally it is item 
A- 3 which may answer the question of what the distribution of vocational^^ 
education "should be in order to meet, the greatest human resource needs. 

National Center for Research 
in Vocational Education 

Section 171 (a) (2) authorizes support for a National Center for Research in 
Vocational Education CNCRVE) , whose responsibilities are to: 

(A) Conduct applied research and development on problems of national 
significance in vocational education; 

(D) Develop and provide information to 'facilitate national planning and 
policy development in vocational education; 

(F) Work with states, local education agencies, and other public agencies 
in developing methods of evaluating programs, including the follow-up 
studies of program completers and leavers required by Section 112, so 
that these agencies can offer job training programs which are -more 
closely related to the types of jobs available in their communities, 
, regions, and States. 

The Center for Vocational Education, The Ohio State University, was designated 
'to serve as the national center. Three projects there have significance for 
impact evaluation. The first is "Examining Vocational Education Outcomes and 
Their Correlates" directed by Robert L. Darcy (1979). Products will include a 
comprehensive listing of vocational education outcomes, an, annotated bibliog-, 
raphy of outcome studies, and a ,. state-of-the-art essay. 

The second, "Interpreting Outcome Measures in Vocational Education," was 
directed by Floyd McKinney (1978) and sponsored by the National Institute of 
Education In August 1978 a conference in Louisville, Kentucky was held to 
explore issues and pitfalls in the interpretation of existing impact measures . 
Among the presented papers is a state-of-the-art review of job satisfaction 
and performance measures (Billings. 1978). A product of the project is a handbook 
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entitled Vocational Education Me asures: Instruments to Survey Former Students 
and Their Emploxers . 

The third is a study of vocational education research and development (R and D) 
product distribution, utilization, and impact directed by William Hull* this 
multi-year study promises to provide data on student and teacher attitudes 
toward the use of R and D products. 

Other Initiatives 

1. The Office of Education is sponsoring^ a one-year survey of students in post- 
secondary schools with occupational programs to analyse their characteristics, 
reasons for selecting the school and programs, and future education and work 
plans (ref. RFP 78-59). 

2. The National Center for Education Statistics, in addition to its work on 
VEDS, is sponsoring further analyses of data from the National Longitudinal ■ 
Study of the Class of 1972, through the late 1976 follow-up, including 
studies of the effects of vocational secondary and postsecondary programs 
(ref, RFP 78-76). 

3. The National Longitudinal Surveys, sponsored by the U,S, Department of Labor 
and directed by Michael E. Borus and Herbert S, Parnes at the Center for 
Human Resource Research, The Ohio State University, has been expanded to 
include a new sample of youth. The new questionnaires that are being de» 
veloped will include improved Items on vocational education and training, 
which will permit new research on impact. 



PROGNOSIS 

What conclusions and implications for impact evaluation in vocational educa- 
tion can be drawn for the future? Vocational educators should recognize the 
need to review their own agency progress in producing the acceptable and 
convincing evidence about vocational education that this paper has described. 

The local reviews in each agency will reveal that some of their ongoing and 
forthcoming work is already consistent with the needs of impact evaluation. 
State and local personnel are in a strategic position to identify impact 
through their own studies and research. The literature search for this paper 
produced a number of ongoing efforts that promise to improve impact evalua- 
tion practices and procedures. For instance, the Washington State Advisory 
Council (1975) identified useful measures of success based on surveys of em- 
ployers , students , graduates, and local advisory committee members. Other 
promising efforts for improving impact evaluation practices have also been 
reported (Franken and Earnhart, 1976; Hamlin and Muth, 1977; Illinois State 
Board of Vocat lonal nducat i on and Rehabilitation, 1974; New Educational Direc- 
tions, Inc., 1975; and Riverside Research Institute, 1971 through 1974). 
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Designing and performing impact evaluation will lead to better understanding of 
the purposes of vocational education programs. Cooley and Lohncs (1976) suggest 
that the primary value of education programs rests not only on immediate out- 
comes but also on the potential for transfer value over the long run. Attempt- 
ing to evaluate impact in such terms requires study that may clarify the role 
of the program in the first place. 

Taking advantage of the need and opportunity for impact evaluation is far easier 
to discuss than to perform. Because of limited resources, vocational educators 
' must conceptualize their practice of impact evaluation over the short- and long- 
term, and utilize opportunities for cooperation. Cooperation between state 
and federal agencies engaged in impact evaluation, is necessary, and may be 
achieved through mechanisms such as the National Coordinating Committee on 
Research in Vocational i-ducation. Cooperation should be encouraged in idonti= 
fying sources of suppUmentary funds to support the design and performance 
of promising impact evaluation proposals, especially those developed at state 
or local levels. Federal agencies should recognize the value of such coopera- 
tive proposals. 

An analogous need for cooperation between agencies,, and advisory councils exists 
within the states. State or local agencies should require that researchers con- 
ducting evaluation studies within the state clarify the potential of the study 
for impact evaluation. In allocating available funds for impact evaluation, 
consideration needs to be given to the development of long-term impact evalua- 
tion capabilities. Research coordinating units, university researchers, and 
others should recognize. the state level need to develop such capabilities over 
the long run. 

As impact evaluation is conducted, information relating to its completion 
should be disseminated as widely as possible. The implementation of systematic 
procedures would also produce benefits. Copies of impact evaluation materials 
and reports that explain procedures and discuss findings and implications for 
the future should be sent to the National Advisory Council and should be 
contributed to ERIC for retrieval by interested persons in all states. More- 
over, the Bureau of Occupational and Adult Education, the National Advisory 
Council, and the National Center for Research in Vocational Education should 
consider undertaking an ongoing, systematic review of the state-of-the-art in 
impact evaluation. At present it is possible that impact evaluation conducted 
or sponsored by agencies in vocational education are never revealed either to 
the National Advisory Council or the Bureau of Occupational and Adult Education. 
As a consequence, the potential for serving accountability needs has been lost. 
Moreover, the potential for one state to benefit from knowledge about the 
feasibility, methodology, and usefulness of impact evaluations completed m 
other states is never achieved. 

Action on the part of relevant agencies and groups to implement these proposals 
would advance the state-of-the-art and achieve more effective and useful impact 
evaluation. In turn, this would permit realization of the potential of impact 
evaluation in achieving the best possible vocational, education for all students. 
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